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ERROR CORRECTING CODES lj 
1. INTRODUCTION 



With the prominence of "computers in today's • 
'technological society, digijta*! communication systems have 

become widely_„us.ed..in_.aL_yariety. of ^p.pJLLc^ti^ns^_JEpr._.„ 

example, data gathered -by . space pfrobes must be transmitted 

'back to earth, where the information can be processed for 
subsequent use. This example includes satellite "pictures 
that are transmitted and processed digitally to ^obtain 
reconnaissance and scientific information.' Da,ta links 
between^ computers provide enormous gains in computer 
applications: 'Military command and control systems also 

vprovide a broad range of examples. *- 

In digital communications, information is transmitted 
in the form of "binary messages," that is strings ofO's 
and -I's whjch are coded in some , way to convey information. 
For example, in transmitting a satellite photograph,, 
suppose that the Onboard instrument package can'(Tis- 
tinguish 64 gray* levels from white to black, and- thut 
each level is identified by a number from' 0' to 63. The' 
.particular string 101011 / which is. the binary representa- 
tion of the decimal ijumber 43, could be. transmitted to 
indicate that, the light intensity at a particular p^int « 
in t\ie picture, is 'at level 43.' 

You may 'easily' imagine some of the problems that , ' 
arise in digital communication systems. Errors may occur 
in the original erfcodiing of data and in the transmitter. 
Channels may be "noisy ;S^so that* bits are' lost or 
distorted.. Security is often a problem— there may be 
compelling reasons to deny^informatian to some individuals 
whc|*have access to our channels. . 



- . ? 



, 'In this unit we study the problem of correcting- > 
errors in digital; communication systems. ■ Error -correcting 
codes are .developed as 'an , application of linear algebra 
and finite field algebra.' 'The models chosen for .this* 
unit. were, selected for relative ease, of comprehension ; % <? 
more complex models will . be presented in' subsequent '*units . 



v . . 2., THE BINARY SYMMETRIC C HANNEL 

■ ^ — • \ 

S.uppose we wish to send f a binary message through a 
noisy channel which v,may corrupt the message by* changing » 
one or more .of the bits. One" or more zeros may be inverted 
.to' ones, or ones invented to zeros. A simple model'of such 
a channel is called the binary symmetric channel 7 (BSC) , #nd 
is. shown ^tn Figure 1. In this model, a bit is inverted 
with.,probability p, regardless of whether it is a 0 or* a 1, 
and the corruption- of any bit is statistically independent 
of what happens to any other bit. ' We. assume thaf 0 < p < % 



Transmitted Bit n _ Received Bit 




: ) v ■ • 

Figure*!. The binary symmetric channel (BSC). ^Transmitted 
bits are inverted with probability p. V * , 

j < " , 

•We. note that the BSC is not a .good model 'for many 

teal-life channels, although it does reasonably represent 

the errors in s%me computers during, fast data transfers. 

The assumption of statis-tical independence is. valid f*>r 

certain systems in which, encoded data are multiplexed, . 

transmitted, and then demultiplexed before decoding, 



Errors tend to occur in "bursts"— a relatively large X 
number in a. short time peribd, followed by a long -error-* 
free'perdod/ Multiplexing^minimizes> the damaging effects 
of bursts by selecting the successive bits in the 
transmitted signal cyclically from* different codewords. 
^Bursts* of several errqr^ in a relatively short span will 
_\then tend to corrupt bits 'in different codewords instead 
•of/being -concentrated in a single, w>rd or two ami rendering 
thein unrecognizable . (See 'Appendix 2.) » . 

Even though the 'BSC is an appropriate model for only, 
certain real channels, we use it in this module for two 
primary reasons 4 . The first, and more important reason, 
it that t*he BSC affords the mcfst^ elementary discus-sion 
of the theory of-lerror correcting codes available. The 
second is'that despite its limitations, the BSC is 'the 
♦principal^ model by which code performance is evaluated, 
because-it is. the only one for^which algebraic computations 
of code performance are* tractable . Nevertheless, there-' 
are other models that are available and^in current us'e, 
and we d'escribe* one briefly in Appendix 1 . & 

-Of course, it is 'desirable that a message be 
* " » , ' ' • . • 

received correctly with high probability.. For example, 

if tTie bi-t 1 signifies' "by'land" and 0 "by-sea, 11 an 

.*./*! * - / 

error in, trarfsmission could have, serious consequences! y •* 

In our first example we consider- a scheme that substantially 

increases the probability, of* getting a -single bit of 

information through correctly. , 

♦ Example* K (Repetition, .with ."majority/' decoding). 
SuppoSe^e wish % to transmits sifigle'bit of information, 
that is, ,a 0 or* a I* 1 Jnstead of a single 0 we, transmit 
the sequence 00000, and instead of 1 we send Mill. . 
These two repetition strings are the 4 QQf&ewoi/dp in our -code . 
The receive* decodes the message hy ^a.. simple majority vote. 
For example, if 10110 is received*, the. decoder decides by.. ' 
•a 3 to 2 vote that ,11111 was transmitted, whence the ' 
intended, bit was 1. This decoding algorithm mky produce 



the correct result * but, it could also produce an error, * 

which would be the y case here if 00000 had been transmitted 

and the channel, had corrupted the 1st, 3rd and. 4th bits. . 

Under this scheme, the probabi 1 ity • of a decoding e^rroj: is 

w gS.ve*n by: ' , • * 

• « v 
/. %P(E) = probability of 2 bits correct and 3 incorrect ' * 

+ probability of 1 bit 'correct and 4 incorrect 

+ probability. of 0 bits correct and 5 incorrect » 

| a 5 c 2 (i-p)V. + 5 c 1 (i-p)p 4 + p 5 

. p 3 (10(l-p) 2 + 5(l-p")p + p 2 ) . ' . 



3 ^ 2" 
P (6p ■ 



15p + 10) . 



{ , If we had simply transmitted the intended bit instead 
q£ using a code, th e ^probabi lity of an error would have 
been p. Using the co^e we have a new probability of error 
P(H), which depends on the original p. We tabulate a 
few values to show how these probabilities compare: 





0.0£ 


0.10 


0.15 


D.20 


0.25 
— . — lt- 


0.30 


> 0.35 


0.40 


0.45 


P(E) 


0.002' ' 


^p.bi 


0.03 


0.06 


0.10 


0.16 


-0.24 


0.32 ' 


0.41 



The table shdws, the gain we have made »in reducing the 
probability of an 'error — by a factor of 10 "when p = 0.1, 
for example. But we*<have paid* a price for this improvement 

Por.to transmit, one bit'of information we now need a block 

* • \ . • 

of* f ive # £its . In this case we would say that the informa- 
tion rate , or number of information bits divided, hyp the 
number of message bits, is 1?5. We note also that our "code 
will'correct as many as* 2 errors in a message.""" We have 
designed a 2-error correcting binary repetition code of 
block 'length 5, in whiph there are two possible codewords, 
00000 and 11111, . 



Exercises 



1. For a i-error corre^ng binary repetition code of' block length 
3, with cadewords OOp^s 

a. «find a formula fot/ P(E) in terms of p, aS was done in 



Example 1; ^ 



s 



b»* compare the values of P(E) and .p for p ■ 0.05, 0;1, 0.15, 
0.2, 0.25, 0.3, 0.35, 0.4* and 0.45; % * ^ 

c. find the information rate./^ 

-* * , 



/ 



.3. HAMMING CODES 



* Irt the repetition,, code we used redundancy to 

•increase' the. probability t.hat a message Will 'be interpreted 
correctly. -The general theory of error correcting codes 
addresses the question, "Can" we add redundancy in sueh a 
^ wa*y that? the probability of error will be decreased to an 
acceptable level and the information jatfe will remain ^ s 
relatively high?" (The terms "acceptable" and^rel'ativeiy 
higW 1 are imprecise* the# must be defined by t;he 
- communication svstem designer in light\ Of availabfe 
4 .„ " equipment and tx&nsmiision channel^*) 

_ * In this section* we introduce a clasps ot c{>des in 
~ .which redundancy is added more intelligently than it was . 
. ' in ' the s f m P le repetition code. These codecs, provide a 
" * betteV trade-off balance between the decrease in the 
probability ^ error and the information^ rate loss. 

Also in this section* we. apply some concerts, and 
, results from linear algebra Jto obtain practical error 
■> ' correcting codes. .We perform oun calculations over.- GF(2) ,' 

~. the -field- wi-th-two- elements' 0 and T\ with binary addition 

and multiplication. ■ "If n. is a positive integer, t let 
GFC2), n denote the vecto> space whose elements are the 
iPtuples 'With entries 0 or 1, a*nd whose scalar field is 

erJc : / \ • ' 5 



" ? GF f (2). For example , .with n = 4, the vectors J (l ,0, 1 , 1) 
'and (0,0,0,1) ^are two of the sixteen elements of GF(^ 4 . 

We may now offer a formal definition , of a* concept , 
, • - m 

which we have already used loosely in describing the 
** 

simple repeti^on code. 

* Definition . A binary % bode of block lengthen is simply j 
a' subset C of GF(2) n . Each element of C is callVd a 

. * fx « 

codeword of the 'code. " ^ 



Elements^ of GF'(-2) n , including codewords, may be / 
£ei 
s, 1 

symbols 



represented as row vectors ,/ as column vector's^ or- as 
"words, 11 '' which Are strings ofV^s or l*s. Thus, .the 



(i,o,i f io, 



and It) 11 



represent' tfce same elemen.t of GI^(2) 4 . 'We shal 1- be "some - 
what cavalier in pa'ssing between these representations, 
^nd between the terms "vector 11 and "word." 



An early impetus ^to coding theory was providfjd by\ * 
, Richard W. Hamming,* when he developed what are now palled 
Hamming codes.. These .are binary^odes of block length/ 
2 m -l, where m is a positive* integer. In Examples 3/iand*3 . 
we describe the code and a corresponding decoding / 
algorithm* for the case* m = 3. # * / * 

• , ' <* 

"Example 2 (A. (7,4) Hamming code**). To obtain a 
(7,4) Hamm ipg code 'C, we first form the 3'x (2 --1), or 3*7 
irfatrix w)iose columns are- the binar.)* representations of the 



*R.W. Hamming did much of his pioneering work on coding theory 
at ^he Bell ♦Telephony Laboratory, Murray Hill, New Jersey., He is * 
currently a Professor of Computer Science at the Naval Postgraduate 
School, Monterey, California^ * 

**The indefinite article is used because any permutation of -the** . 
columns in the fundamental matrix H in'O.l) yields 'another (7,4). • 
Hamming code. * ✓ . ♦ * * *v ■ ■ * 



integers from-.l to 7. Since these representations are, 
in order, 001, 010, Oil, 100, 101, 110, 111, this* matrix , 
• which is called" the parity check matrix of the code, is: * 



(3.1) 



H = 



0 , 1 

1 0 
1 0 



A quick inspection reveals ^:hat the rank of H over 
GF(2) is 3. Hence, the hu^fisnace of H is a 4-dimensional * 
su{>space of GF(2) 7 (see Exercised), and this nullspace 
is. the set- we t ake- for C. Thus, a vector x in GF(2)^ is a 
codeword if and -only if x satisfies the condition = 0 
(here 0 denotes -a zero~vector) , when x is represented as a 
column vector. For'example , if 



x = 



n/si 



T 




"1" 


0 




0 


X 




0 


0 


and y = 


1 


1 




1 


a 




1 


i\ 




r 


"0" 




"a" 


0 


and Hy = 


0 


0 


l 




then/ since> 
Hx 



^we have xcC-and ytfC.. That is, x is a codeword and y is noS 

Since is a 4-^djnensi9nal. vector space over GF(2), 
Ih&e ire precisely" 2?% 16 codewords in the (7,4) Hamming 
code. *So far we have -seen* only "one elemenj^of fc.,(the 
* ; vector x that, as a wor.d, is represented by the, string 
lOroiOl) iii thetlasrt Sisplay.. But we know that QOboOOO ,£s .* 
also<in^C, since HO = 0 by elementary 4 properties of matrix-' 
multiplication. , In Exercise 3' you will be challenged to 
~'nd' kll this, c^ewords* in 'this code. . " %• 



In Example 3 below we present a decoding , algorithm 
f or -the £7,4) Hamming code. of Example 2., In,..tfij.s code^ r 
* %four"*6f the^seven bits in- a codeword may be^» designated as' 
* the information bits. The -remaining three 'are/qa'l led, the 

, ' 12 . - ' ? 




parity check bits 'a3&, simply, , the check bits. In this ' 
case the information ^rate is 4/7, For a more general 
Hamming co4e, <with m any positive integer, tfte parity 
check matrix H wifl hSve dimensions mx (2 m ~l) and rank 
2 m -m-l,-and the information rate will be ( 2 m -m-l) /(2 m -i) . 

Before we present Example 3, we o^ffer several exer- 
cises a^nd'some additional discussion designed to get you 
better acquainted wi^th the (7^4) Hamming codes. This 4 
material will help you understand the decoding algorithm 
Example 3 . * - 1 . 



injl 



Exercises 



Consider the following elements of GF(2) (regarded as column 
vectors) : 



1 




Y 




o" 








1 




0 




1 








1 




0 




0 








0 
0 


y ^2" = 


1 
1 




1 

0 


> 


\ = 




0 




0 




1 








0 




0 




0 






1 



3. 



b. 



Show that for i = 1,2,3,4, x ± is a codeword in the (7, A) 
Hamming code of Example 2. (That is, 'show that each'x^C, 
the nullspace of the matrix H in (3.1) y) 



Show that the set {x^x^x^x^} is linearly independent? in 
the vector space GF(2) 7 ove» the scalar field GF(2). ; - (Hence 
• the subspace C has dimension 4; this agrees with our earlier 
ob^ery^Ltion that the rank of H is 3.) * 1 

Use the result of Exercise 2 to list 
(7,4) Hamming code. <. • 



all 16 codewords in our . 



. Although we now/have^ar- list of tfieT codewords , the 
approach -used to offiEttLxT'it ,in Exercises 2; arid 3 rs not ~ ' 
particularly instructive. ' We offers better technique 
Using elementary matrix manipulation. /We first introduce 



two additional 3 x 7 f ma trices : 
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'0 

* 


0 


0 


1 


: o 


0 


0" 




"0 


0 


0/0 ; 


1 


1 


1" 


0 


i 


1 


0 


I o 


0 


0 




0 


0 


op j 


0 


1 


1 




0 


1 


0 


; o 


0 


0_ 




_0 


0 


0 0 ; 


1 


0 


1_ 



Theivvby'jpatrix addition we have 
H * H x + H 2 , , , 



an4^4r any element x. (represented as a column vector) in 
GF^I we .have \ 

TK^tefore», if xeC, the ^nullspaCe of H, then * 



+ fl 2 x. - 0 , 



f 



$t, since we are calculating in GF(2) , we Have 



2) 

Or.'-.-* - * 

•,K^w let Xq,x 1 ,.v.,x 6 be the coordinates of x. Then, 

'because of the blocks of zeros in'H..,H 9 , we obtain the 

- fpliowiii^ reduc^i< 



H^x, = H 2 x. 



Lions: 



(3.3) 



H : x = 



0 0 

0 110 
IOIO' 



H 2 x 



111 

0 11 

1 0 1 



Substitute (3.3) into (3. 2/ to obtain 

- • r * * , fx 



(3?4) 



"r 


1 4 




\' 




'0 


0 


0 


1" 


0 


1 1 




x 5 




0 


1 


1 


0 


1 


0 1 




** 




1 


0 


-1 


0_ 



v 0 



Equation (3.4) can be solved to obtain * 4 ,x 5 ,x 6 in 
Q :ertas of X^x.. ,x 7 ,x, by a matrix inversion. Since - 9 

ERJC - • . Vj ; ■>■,•■ 



(3.5) 



111 
0 11 
10 1 



1 1 0 

1 0 r 
111 



(see Exercise 4) , we may multiply both si8es of Equation 
(3.4) by thi^sjnaj:rix to obtain 



(3.6) 



"1 


1 


0* 




"0 


0 


0 


1" 


1 


0' 


1 




0 


1 


1 


<r 


1 


1 


1 




1 


0 


1 


0^ 



x l + x 2 + x 3 
x Q + x 2 + x 3 

*0 + x l + x 3 



(after two matrix 
multiplications) . 



Now, the steps used to derive Equation (3.6) are. all 
reversible (see -Exercise 5). Hence, , 



^0 



'for x 



e GF(2) / , < 



(3.7) 



xeC if and^nly if 



A 4 
x 5 
x 6 



x l 




x 2 


+ 


x 3 


<° 


+ 


x 2 


+ 


x 3 


x 0 


+ 


x l 


+ 
> 


x 3- 



We note that x- , x- , x v may be 'taken as the check 
bits as long as the columns i, j r k form linearly inde- 
pendent vectors so that the t x 3 matrix consisting of 
these dolumns is invertible. 0 
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. Exercises . • 

4. Verify that the two Beatrices (3.5) axe indeed mutually inverse. 

v , • > * 

5. Show that *ii; Che coordinates of a vector i'n GF(2) satisfy the 
condition in a(3.7) , then Hx ■ 0. (The verification may be, 
accomplished' by reversing^ the steps, that led Jto Equation (3.6), 
or outperforming the matrix operation needed to find*Hx.) 

t * • 

j>. Again list the 16* codewords (as in Exercise 3), this time using 

the condition .(5*7).^ ' 

* * * * • * 

7. For t^he word x * 1000000 in GF(2) 7 (you may want to represent x 

as a column vector) , 

■\ » ■ *■ 

a. find Hx" and compare Hx with the individual ^Qlurnns of H 
• * « " 
- ' A - (to which column does Hx correspond?); 9 ** 

.b. list the 16 elements in the coset" C + x\ 

8. .Carry out the 'instructions of both parts of Exercise 7 for the - 
vorp\ y * 0100000. 1 . - 

— ^ : : : — 

By now wfe have a good understanding of the space C. 
We 'know thW of the 2 7 = li8 elements in GF(2^ 7 , exactly 
16 are codewords; we know which ones they are, -and* we have 
a way to identify them (Condition (3.7)).. We also have a 
good feeling for the coset structure: we know that thejpe 
are eight cosets, each with* 16 elements, and' that the 

/elements b^atry'one of the cosets other than C itself 
?can be obtained by choosing a fixe'd bit (or coordinate) 
position and inverting the bit in that position,/ for each 

* element of C. (In Exercise T'vie chose the first coordinate 
in Exercise -8 the second..) ,„* * , 

We call special .attention to Condition (3,7). This 
condition shows- that- while four -of the seven bits in a 
codeword may be assignecPas we wish (here the indicated 
brt^are x Q> x^ ,x 2> x 3 ) , there is no choice y 1 the remaining 
three. For if we-assign values to X Q , x 2 , x^ y% then 
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the values of x^ , x^, x^ are then predetermined by (3.7), 
■Thus, each 7-bit codeword contains ont^ 4 information 
bits, which confirms our earlier observation that the 
information rate is 4/7. , 

We now present a decoding algorithm for our (7,4) 
Hamming code.* This algorithm is called a maximuti likelihood 
decoder, although the reason for^this name will not become 
apparent until t v he discussion after the example. 

Example 3 (Maximum likelihood decoder). SuppoSe that 

a dertain 7 -bit codeword c from our (7,4) Hamming cftde C 
7 * 

in GF(2) is transmitted over a noisy channel, and that ithe 
word r = 1001010 is received at the other end. If we 
represent r as a column vector, we may calculate Hr, for 
H the parity checfc' matrix (5.1): 







T 














0 














0 




"0" 




*0" 


(3.8) ■ 


Hr = H 


1 




1 




0 






0 




1 




0 






1 

0 











Since Jlr'?* 0, the message recipient knows fhat r i C, and 
hence r f c, and therefore an ^pfor has been made in * , 
transmission. m w hat word was originally sent?" ^ 

** Suppose we represent the received word r as the 
sum of the transmitted codeword c and an error e: 



{3.9) r = c + e. 

Now let Hr = si then al so *fte^!^Vibecause' 

s = Hr = He + He - He + 0 = He. 

Therefore, while the message .recipient does not know^c or 
e, he or she- does know th^t e is contained in the same 
coset as r, i.e., that e e C + r. The problem of determining- 
c therefore reduces to that of making an "intelligent" . 
choice for e from the coset C + r. 

* ' ' 12 



In seeking the best choice of e in p. + r, we first 
•note that the column vectp* Hr in (3.8) coincides with the 
third column of H. But by elementary properties of matrix 
algebra (see also Exercises 7 and 8), we know aJLso that 
the column yector given by the product 

0 
0 

1 

0 
0 
0 
0 



also equals the third column of H. Hence the word 
0010000 is also in the coset C + r. Moreover, if 
we call the number of 1 1 s in a word the weight of that 
word, then no single coset may contain more than one 
vector of weight one. (This fact can be seen fr^m the 
results of t ExerQises 7 and 8 and the observation 'that , as 
vectors, the columns of H are all distinct.) The word we 
take from the coset C +.r as our choice for the error when 
tjie received word is f - 1001010 is the word *' ■ 

(3*10) e = 0010000. V 

We' now Use Equation"" (3 .9) to find our "best" guess for/the 
original c: 



V 



.c '+ 



-from which (since e + e - 0000000) 



c - r + *e 



= 1001010 +'0010000 J 

= lonoio!;- x 



,We note that our choice of c' can be obtained by* inverting 
the third bit in the received word r. • 

Now suppose 'that instead of a, word- (such £s 1001010) 
from a coset of G, a codeword V 1 had { been received. Then, 



. n" • " ■ 

the error c - c 1 would \be a codeword, so the choice of 

an error would be made xVonuC itself. - In this case^e 

choose the zero word 0000000 for tho error, that is - * we 

assume that no error occurred, so #we take. c 1 * itseTf as 

the choice of c. (The reason for thi*s choice will be 

explained a£±er this example.) -< * 

S ' ' , * ' 

We summarize the procedure above as our? decoding'* 

algorithm for a G7,4) Hamming code. • 1 

Maximum likelihood decoder . For *a (7,4) Hamming* 
code; when a 7-bit word r is received, we: 

• represent r as a 7x1 column vector; 

<t * s 

• find the column vector Hr by^matrix * j- 
multiplication over GF(2) ; 

• test Hr to determine "whether Hr i6 the 
zero vector, tor a column, of H; 

• choose r to be c itself if Hr = 0,\for 
then* r eC; . 

• * choose c to be the codeword obtained from 

r by inverting the j th bit, if ilr is the 
jth column of H. • , % 

l , , i ^ ^_ ! 

Exercises ' { 

9. For the (7,4) Hamming code of Example 2 with maximum. likelihood 

decoder, decode each of the following received words r: 









a. 


.1101011; 




b. 


0011110; 




c. 


4010101; ' 




d. 


.oiooooo H : - ' 






V 


* * 









- /i^Two immediate questions arise in' connection 'with 
Example 3.. The first is: sinCe we are using 7-bit code- 
words .to convey 4 bits of information, which means an 
information rate of* 4/7,, or 0 .'57 whaudo we "gain in 
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trade for thi's 43-percent loss? The* second concerns the, 
name of* the algorithm— why is it caWed the ^'maximum 
likelihood" decoder? 



We begin With the f^rst 4 of these questions^ The 
basic response is that By using the cQde we reduce the" 
probability of error in* conveying each' 4 bits of informa- 
tion. We sftall % measure this gain, as we measjir'ed the 
corresponding gain ih Example 1, but we emphasise- \hat in ^ 
studying gains, a realistic user, must never iose sight of 5 
the losses involved. ; 

l . ' ' • 

For an analysis of the gain ma"de « by. using a (7,4) 

* i 

Hamming code we % use the .-binary symmetric channel (BSCJ 
model*as in Example l.V In this case, however, it-is more, 
convenient to study the increase in the probability of 
"no error," P(N}*, instead of the decrease in the proba- 
bility of error, P(E)." " • * ^ 



If we simply broadcast 4 : bits of informatiotn across- 

it usin*/ a code, then' wfc wntilrl havp ' - 
,4 



a -BSC without using* a code, then w£ would haYe 
\* ?W ■ (1-pJ . Using a (7,4) Hamming code' with 'maximum 

likelihood decode*, we make no error grecisely'when » " ^ 
. either one or none of the 7 transmitted bits is* corrupted.- " 
1 In this- -case we have . * \ 

(3.11) P(N) "probability of ' 7 bits correct : and 0 incorrect** 

. t ' + probability of 6 bits Correct and X "incorrect , *\ 

> I - > - d-p3v^ ;7 c 6 (i- P ) 6 p\ ! 

. -„.'.•" 11 , •-.a-p) 4 (i4'p-iip 2 +6p 3 ). . ' . ^ -vii 

* ** JiruV^the polynomial f (p) = l" + ,4p - lip 2 + 6p-. provides/* -* ' 
a' measure of the gain in the probability o£ no» ejbr<3r* *f ■ 

.Since £(0) f ^(l/**! = "V' aildl-an easy analysis of the 
*. .derivative shpw§ thaff'(p) > 0 for 0 < p < 2/9. ah<i,^* 
« f'(p) < O.for 2/9 < p ;< i; our best percentage gain^in 
rae- probability of no >r*ror v oc curs'" VtiD 4 *2/9. SincV.% \ 
f(.2/9) « 1.41, this, maximum. gain is around 41 percent*- * v-, 

^ERJC ' > „ ^ . 



We tabulate the values'of (1-jT) , P(1J),' and the percentage 
gain for severalr values of p. * v % 



TABLE 1 

Percent, Gain in the Probability'of.No Error 
(for selected values of p) 











r 






• • 


* p i 


(1-P) 4 

—-^ 




Pefcent Gain 
(f(p)-l)^ 


o m • 




— f— 

J- . uu 


* 

4. 


. o;os 


0.81 


0.96' 


17 


U.1U 


\J , 00 

> 

0.52 ' 


. 0.85 


30 


; 0. 15 


0.72 


37 


'0.20 


0.^1 


0.58 


" 41 


; " 2/9 


0.37 


0.52 




0.25 


0.32 


0.44 


*1 


0.30 


0.2^» 


0.33 


38 v 










0.35, 


0.18 


0.*3 


31 


0.A0 


0.13 


0.16 


'22 


0.A5 


0.09- 

/ 


„ 0.10 


12 



.jrate. 



t TKe table indicates that this coding "scheme may v,e^y 

\. welljbe appealing, depdn|ding on requirements for "accuracy 

^ an^other factors, suchT as constraints*on tne information" 

Mn addition* it ma/ be that thi'ssg sch.eme would be - 

$j^*de$irable for values of p that clo hot yield maximum 

percentage gain in the probability 'of no error. For 

example,* if a Jfigk degree of accuracy is required, <hen 

the increase iji^che plrobability of no error from 0.96r to -* 

0-99'S Crounde'd £o* l s 00 in the table) for p = 0.01 could 

\>e attractive to the iiser, and well worth the trade in the 

information rate/" . - / 

• V ' \ > . - * I 

- The second question indicated above also has an 

jinVeresiting answeisO^ In the decoding algorithm of Example 
*'ther« is su test to made, namely a determination *of 

• t O 1 W 



whether, •the' received word jr is a codeword or not. Lf 
r eC, then the error vector, e^is also in C, and the basis' 
of oUr choice of e 8 0 for the error is that under' certain ' 
-coi\ditdons this choice maximizes the probability that the 
chosen^yector is the errop vector; Similarly, if r/C, 
* the clioi.ce of the vector of weight one ffom the.coset 
C + rHmaximizes. the probability *of the error vector (again, 
unde^'certain conditions) . 

L§t us explore these probabilities, again using the 
BSC mocf^el. When^we transmit a 7-bit word over 3. noisy 
channel* %he error vector component is 0* *if the bit is 
correctly received, and 1 if the bit is. inverted in the 
channel. Thus, the error vector may be regarded as the 
outcome of 7 repeated Bernoulli trials with probability 
p of a 1 at' each £rial and 1-p of a- 0 . Hence, for a an 
integer with 0 £»a £ 7, the' probability that the^ error' 
vector will have weight a (i.e.*, a entries 1 and 7- a 
entries 0) is binomial: $ 

(3.12) P(a x ) - - 7 C a p a (l-p). 7,a , a - 0,1,2,.'. ;,¥. 

Thus, the probability of any error vector depejtf&s on 
i£s* -weight . When Vie G, we seek the vector in C which 
maximizes (3,12J. We note *that .the zerQ vector is in C, 
„ and* that no vector oW weight 1 is in C. £ut C cannot .have 
a Vector of weight 2 either; for example., if 1100000 e C* * 
then we wo'uld ha,ve,* r 





"1" 




Y 


t 






rii 












1 




0 




i 




0 




1 








0 




0 




0 




0 




0 






0 = H 


0 


= H 


0 


+ H 


0 


= H 


0 


- H 


0 


y 






0 




0 


0 




0 




0 








0 




0 




0 




0 




0 




( 




0 




0 




0 




0 




0 







.wftich would imply that the words* 1000000 and^0100000 are in 

, the" same cose,t. • Since 1111111 e C, we see that no vector of 

weight S of 6 can be a codeword^ either /.*.« (These facts are 

. • v. ' 

also known from the results of Exercises 3 and 6.) Thus,* 
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to, maximize the function P(a) in (3.12), we neecf consider 
only a = 0,3,4/7. " But since 0 < p < 1/2, we need 
consider only 0 and 3. From (3.12) we have .» 

P(0) = (1-p) 7 , P(3) ^^Sfl-p) 4 ? 3 - 
e, t. ♦*-''♦» 

■ mi a d-pr ■ • - . ■ 

v 

and (l-p) 3 /35p 3 > 1 when 0 < p < 1/(1 + ^3?), or 0.<p~< 0.234 
(approximately). Thus, when the 9 received word r is a 
codeword, i.e., when r t C-, the choice of e = 0000000 for 
the error (with the* concomitant choice of r for c) is the 
choice thar maximizes the probability that .the chosen 
vector is indeed the error;* provided only that 0<p< 0.234 . 
In practice, real-world digital communication channels 
satisfy this modest requirement, with* room to spare. 

- s When N the received wor,d r jl s not a codeword, we would 
like to know the condition(s) under which the choice of 
the vector, of weight 1 from C + r maximizes (3.12). Since 
C + r has vectors of weights 1 , 2, 3,4 , 5 ^and 6 only, and 
since 0 < p < 1/2, we must, maximize P(a-) in (3.12) for 
a 1,2,3, ^only. Erom (3.12*1 we have 

Hi) - 7(1-P) 6 p;-* *P(2) 21(l-p) 5 p 2 ; P(3) = 3^(l-p) 4 p 3 .\ 

•**£t follows that . • . ' % ' , 

P(l) >.P(2*) whenever 0 < p < 1/4,- and * ; 

jP(l) > P(3) whenever 0 < p < 0.309 (approximately). * 

The- verification is similar to tjie one abotfe for^the case 

r e C. Therefore, in alj. cases the algorithm in Example 3 

calls for the choice of e that, will maximize P(a) in (3.12) 

whenever 0* < p < 0. 234 (approximately); hence the name/ 

"maximum likelihood' decoder . ■ 
i 
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■ - / . 

We note that if the channel noise causes. 1 error in 
/-ff 7-bit message, i.e., invents 1 bit, then the decoding 
algorithm will correct the errar. Thus, in Examples 2 and * 
3 wejiave a l-error correcting binary code of block length 
7/ with 16 possible codewords. 

We close this section with an exercise "that represents 
aA invitation to share the experience of working through a 
Hamming code model from the beginning. ' -This model cor,- • 
responds to the value 4 of the parameter. m . (we considered 
the case m = 3 above). No doubt you -will find that the 
calculations rapidly become tedious/ as m increases,. Larger 
models^re best handled by computers . N S~ x \ 

i ~ r ^ ' / . / 

Exercises ~ * * / J 

10, For the (15,11) Hamming code with columns of the parity check 
matrix in natural order, and with maxftum likelihood decoder: 

a. find the parity cn4ck matrix H; C 

b. -find the information rate; " \ 

c. decode the received word 111100101100010; 

d* find conditions, corresponding to (3.7) that characterize 
codewords in GF(2) 15 ; * ♦ ' > 



e. carry out an analysis to compare the gain in the 

probability of conveying information correctly under this 
~ coding scheme with t.he corresponding probability of no 
error under no coding scheme, which i*/ (1-p) 11 . 



' " . 4. MODEL EXAMINATION . 

For a 3-error* correcting binary repetition code of 
block length 7 V with codewords lllllll and 0000000, • 

a. find* the* information .rate; " 

b. decode the received word 0011000; 

c. find the probability of arwerror P(Ef,*<using 

the BSC model, * - 19 
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2. Using our (7,4) Hamming code with maximum likelihood 
decoder,' decode the words: ' 

a. r = 0110011; 

b. r = 1101101. 

In problems 3 and 4 you will need" results from 
Exercise 10. t * 

3. Using the (15,11) Hamming code of Exercise 10, with 
maximum likelihood decoder, decode the words: 

a. r = 101010101011111; 

r 

, b. r - 110111000101101. ' " 
• " • * * 

4. In* our (7,4) Hamming code we found the fallowing 
distribution in C: 1 word of weight 0, 7 of weight 3,' 

7 of weight 4, 1 of weight 7, and 0 of weights 1,2,5,6. * 
In the (15,11) Hamming code, find: 

a. the number of words of weight 0- * 

c 

b. . the number 4f words of Weight 1; . 
^ c. the number of words of weight 2; 

d.^at least one word of weight -3 (there are 24- of them).; 
- e.- the total number of codewords. 



5. ) SOLUTIONS TO EXE RCISES" 

5 : , • 

P(E) = probability of 1 bit* correct and 2' incorrect .-. 
-■ + probability of 0 bits correct ^xs^S.^toix^t- \ 

-' 3 c^(i-p) P 2 + P 3 > p 2 (3^ P K*: v 'V \y r j'% : ; ' : 



b. 





0 ,05 - 


oaf 








' "i 










0.OQ7. 


0,028- 


b:o6i- 


>104 













c. ■ 1/3. 



a. The verification can b£ accomplished \by carrying out the 
matrix multi 
For example, 



matrix multiplication to show that Hx i ■ 0 for i = 1,2,3,4. 



Hx 



1 , 



0 0 0 1 1 1 r 

0 110 0 11 

1 0 1 0 1 O'l 



o+o+o+o+o+o+o 

0+1+1+0+O+OfO 
H 0+1 + 0 + 0 + 0 + 0 



b. Suppose that a linear combination of them's vanishes, i.e., 
that + a 2 x^ + + = 0, (Note that the d's 'are 
all j0 or Vand .that all- arithmetic is carried out in GFj(2) .) 
Then from the first and second rows we have 



0, and 



vllpon adding we obtain c£ 2 + o^fj^so (* 2 = o:^. The third 
row- shows ' that ,0^ - c^, the fifththat a 2 = and the 
seventh that « 0. Thus, a ± = 0, i = 1,2,3,4, so 
x^x^yXyX^ are linearly independent. 



We.may"f*nd all elements in C by ^ormjng all possible ^linear 
combinations + o^x^ + '°3 X 3 + °4 X 4» wnere tne a i* assume •/ 

the-value% 0,1,. 'We list .th'e-16 elements ^in C as* words: 

0000000 r ' 1110000/ 1001100, 0101010, . 0010110, 0100101, lQOOOll, 

*ooiioq1\ oobiiu, 0110011-, . 1010101, noiooi w ioiioio/ 0111100, 
nooiio, num. 

Verification* can be accomplished by "cajr^rying out the indicated 
•matrix multiplication to obtain the identity matrix. 

We perforo'the Suggested matrix multiplication: * . 



0 0 0 1 1 1 1 
0 110 0 11 
10 10 10 1 



x^O 



x Q -hc/-hc 3 



I 

6.- The answers, which* "are given in the solution for Exercise 3, can 
* - / * 

be obtained by making all possible assignments for Xq,x^,x 2 ,x 3> 



x 3 +x 1 +x 2 +x 3 |bc 0 +x 2 -hc 3 +x 0 +x 1 +Xj 
x^+x 2 +Xq+x 2 +x 3 +Xq+x^+x^ ^ 

X 0 +X 2 +X l +X 2 +X 3 +X 0 +X l +X 3 



we take x 0 



1, x « 1, 



then allying • For example,, if 

x 2 = lV x 3 * 0, .peiTx^," 1 + 1+ 0-0, x 5 =1 + 1 + 0 = 0, 
-1 + 1 + 0= 6, aig^we obtain the codeword 1110000. 



A 6 
a. Hx 



, which corresponds to the fitst column -of H. 



b» The words -in C + x can be obtained from the codewords by 
* inverting the first bit in each word. Referring to the" 
■ solution for Exercise 3 we obtain the 16 words in C +. x: 

% ioooooo, oiioopo f , ooonoo, iioioio, loioho, 1100101, 

0Q000U, 1011001, 1001111, lU06ll, 0010101, 0101001, 

r oonoio, iiinoo, oiooiio,. olmii. 



,8. a. Hy 



, which corresponds to the second cplumn of H. 



b. The words" in C + y can- be obtained from the codewords by^ * 
inverting the secondjbit in each word: 

0100600, loioboo, 1101100, booioio, -0110110, 0000101, 
% nqgoii, oinooi, oid^m, ooioon, mioioi, looiboi;' 
. ♦ iinoio, ' ooiiioo, loooiio, loiiiii. / v %7 
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9. a. 1101001 

b. 0010110 

c. • 1010101- . 

d. ooooooa 



10, 



00000001 1 A 1 11111 
0 0*0 111100001 111 

0 i i'o o i i oto i i o o 1 1 

1 0 1 0 1 0 1 0 1 0 1 0 1 O'l 

b. 11/15. 

c. 111100101100000. N r 

d. Some care must be taken in using the technique that resulted 
in Condition (3.7). For example, if we try to echo the 
method to find tfce last 4 variables in terms of the first 11, 
we obtain a singular matrix.. We must select a set of 



unknowns with a matrix of rank 4. 
of which one is the following: 



There are many choices, 



< 


00000001 


0 


0 


0 0 


! 


i 




oooiiiio 


0 


0 


°.° 


! 1 1 


i 




oil. o oiio 


0 


0 


o-q 


1 0 1 


i 




1 0 1*0 10 10 


0 o 


P 0 


! 1 0 


1 




0 0 0 Q 0<0 0 0 


1 


1 


1 1 


1 .0 0 


o" 


m 


ooo q,oo^o o 


0 


0 


0 1. 


[ 0 0 


0 




ooo cf o o^o b 


0 


1 


1 0 


! o o 


0 




0 0 0 0 0,0 0 0 


1 


0 


1 0 


I ° 9 


0_ 



•If x is a codeword then, as in Equation (3*2) , when we 
, represent, x as a cplumn vector we find H.x~* H 0 xl If we 

' A ^ ' 1 , L ' ' 

' denote, the bits (or the coordinates) of x by .x^»x^, . . . ,x^ 3 
we abtaJLn a re stilt similar to 1 Equation (3.4) i 3* 
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1111 
0 0 0 1 

0 110 

1 0 1 .0 



*10 



00000001111 
00011110 It 11 

oi r 00110011 

1010 1*3 10101 



12 
C 13 
C 14 



Since 

i i li 

0 0 0 1 

0 110 

1 04-0 



-1 



1110 
11,0 1 

11^11 

0 10 0 



we obtain 



X 8 




0 


1 




1*0 0 1 0 


1 


1 


x 9 


S3 


1 


0 


1 1 


0*1 b i i 


0 


1 


x 10 




1 


1 


0 1 


0*0 I'll 


1 


0 


*11 




p 


0 


01 


1 1"1 0 1 


1 


1 



"6 

*J 

x 12 

X 13 

x 14 



Therefore, x is a codeword if and only if 

x rt L z J 74 "7 "13 " 

ir 



*3 



x x + x 2 + x 3 + x 4 + x ? + x 1? + x 14 



x 0 + x 2 +x 3 .+ x 5 + x 7 + x 12 + x 14 , 

x 0 + x l + V +x 6 + x 7 + x 12 + x 13\ 
*11 " x 3 * : x 4 + x 5 + *6 + *12 + x 13 + "14 



10 
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e. Using the code, we have for the probability of no error;' 

P(N) « probability of 0 bits incorrect and 15 correct 
+ probability of 1 bit incorrect and 14 correct 

.« (1-p) 15 +>5(l-p) 14 p, > 

* jU-p) n (l + lip - 3?p 2 + '41p 3 . - 14p 4 ). 

The following table shows the gain for a few values of p: 



p 


,0.001 . 


0.005 


0.010 


. 0/05 




' 0.989 , 


0.946. 


0.895 V 


0.569 \ 


P(N) 


0.99990 


0.9975 


\ i 

0.990 


0.829 



6. -ANSWERS TO MODEL EXAMINATION 

a. ,1/7; ^ . 

b. 0000000; |> 4 (35 - 84p + 70p 2 ^- 20p 3 ). 

a. 0110011; 

b. 11O1001. 



a. lllOlOlOlOlllll^r- 

b. 110111000101101. 



a. 1; 

b. 0; 



i 



*c» 0; • ' - ■ 

. Use- the-xesult^ot^xerciae JLOdJ^Jfer^example, if^we assign 
. * x Q M, and ~b for k - 1/2,3^4 , 5 f '£, 7 ,12,l£,14s jthen 

«-0y x 9 'x^ Q - I, ■ arid. we obtain the codeword 

lOOOOOOOOllOOOO; * - 

- - a * ' * 

1.. .2**, or 2,048.. «.\* ; ' ' 

i : : >••' ^> W?' ■ K - ... 
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APPENDIX 1 
AN ALTERNATE TO THE BSC • 



We describe briefly a model that is applicable 
some situations where the BSC model fails. Note first 
that' the gSC model may be represented as a finite Markov 
chain witj/^two statesman "^rror state 1 * E fc and a Veorrech: 
state 11 C, with matrix oi; transition probabilities g^ren by 



C 

1-P ' 



E . 
P 

1-P. 



A graph-theoretic *model is also useful in visualizing 
this chain%- vThe states may ,be represented as vertices, 
and the transition probabilities are shown on the edges. 




1-F 



The Markov 'chain approach can| be used to obtain a 

model which represents reasonably *well some channels in 

which errors occur in bursts, far example, certain HF 

* • . * x ' ' . * *. 

channels.. In this model there are thtee states: C n is - 

v> 

a !, guard state," and, -the transmission is ^xror free while 
the system is in this state; the ( remaining sta,tes and 
E are "bursty 11 (noisy) states. The matrix of transition 
probabilities is given by * 



tfjiis chain mdy also be represented as a graph. 




This model can be interpreted in a straightforward 
way. For example, suppose that the system is in state 
Cg, so that error-free, transmission v is in progress. If 
p 3 is small, then there. -is a rfigh probability of continuing 
in state C G ', but it is also possible to make an error. We 
move to the error state E with probability p 3 , and from 
this state it is possible to continue in state E, return 
to state C G , or enter the error-free state C^. Analyti- • 
cally, the' difference between states- C G *and C N *is :that p 3 
is much*smaller than p^ Thus when the* channel is in state 
;X) G , 'it has a, tendency to remain there;. , When .the error 
( burst is over, we move back to state "C^ for another period 
of errorrfree transmission* 

' * x 4 

x "This 3,- state Markov model can be generalized to a 
,model with 'any number- of states'. Increasing ^the number bf' 
;stttte^t?o^5i say, witK"T3iree"^rbr^ - — 

error states may t enable, us to* model a wider class of 
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channels accurately. However, there lare two major 
difficulties. First, as the number of states increases, 
the physical significance of these states may be difficult 
to ascertain* audi, second r even in the case of the 3-stat6 
model, computation of P(N) in Table 1 in terms of p^ p 2 , 
p 3 , q^ is quite complicated. 

The customary method for constructing an analog to 
Tsble 1 for a multi-state Markov model is to use a Monte 
Carlo approach, that is, to generate an n e,rror stream" of 
0 f s and l's according to the model. The error stream is 
then divided into n-bit blocks and decoded. If decoding 
produces* any. result other than 00... 0, the all-0 block, 
.then a decoding v error has occurred. 

■ ' It is worthwhile to note~ that if errQrs tend to occur 
in bursts, 'then a code, such -as a Hamming code, wfiich is 
designed to correct single, isolated errors may be worse 
than useless, for two reasons. First, most of the time, 
while the channel is in the guard state, no errors occur 
at all, so that all the code is doing in this state is 
adding unnecessary redundancy; and second, when errors do 
v occur, the cnances are that any block with at # least one. 
error will contain more than one, which results in a de- 
coding error, thus producing even more errors. Therefore, 
channel considerations are very important in designing 
error correcting codes. Our nexf two modules on error- 
correcting codes will contain some examples* of codes that 
are -designed to correct burst errors.' 
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V„ ' tf^smit tte^aiZifm^f ^fN^^ " . 

. .The'r^cxpi^nt of the. .^i^n's^s^S^^^ej^^^ -^r , 

v -y <~ £5ll» r 2;L* r 3i>* *" > ^ # ■ 

* . * by .aoJ.umn into an ra x'n arjray ^^.^j^V ^Jhe rows of R *" ■ 

7^ ard^Cpossibly) corrupted, codeWoxdsi" \f£ : "£ £6urs*t!V tff :< • ~\ - V. 
V ©wors <jf length b'.< ra has occurred in transmission-over/ ' 

" «; the! channel ,^the. errors axe spread through successive -rows * 



4 / 'x;? ; .' .p£,R. ■ .Therefore,. iiPa g iyen ^ row , : what eyjer errors occur v 
*i# V * e . 4< J° ^ e isolate*** independent -errors ^ In such a-situar 
v; r '^V4tiqn^. the .binary symmetric channel -irtay- give aV v r'ea$onatfIV 
l . \ model for the errors appearing in; the -received- words to be 



V 



* -decoded. 
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student' form l 

Request for Help^ 



Return to: 
EtiC/UMAP 4 
55 s Chapel St* 
Newton, MA 02160 



iv -- -Student : ' If you have /trouble with a specific part of this unit, please fill 
r---^ u ?-*?4 s and, take it to your instructor for assistance. 'The information 
4^^l£j$^ ti * 1 l * e lFU the author to revise the unit , ' \ 
' Your Nam,e. *'~ - 



- Page 



\ O Upper - 
OMiddle 
O Lower - 



OR 




Unit No. 



or- 



Description of Difficulty: (Piease be specific) 



Model Exam 
/Problem Np. 

Text 
Problem No. 



4* 




Instructor : Please indicate your resolution (of the difficulty in this box. 
i Corrected terrors in materials. List corrections here: 



o 



Gave student better explanation, example K or procedure tjian in unit; 
Give brief outline of. your a'ddition here: ' . *./ s 



V Assisted stydent in acquiring general learning and problem-solving 
skills (not using examples -from this unit.). — < c \ . _ 



i 



Instructor*' s Signature «_ 



'ERIC 



:,JPleas€t use reverse if necessary^. 



. < Return tq: - • 

• STUDENT FOKM 2 [ EDC/UMAP 

Unit Questionnaire * " ? hape ;L S Joi,;n 

i ^ / Newton, MA 02160 



Name 40 



Institution * a Course No, 



Unit No^ Date 



Chfeck the choice for each question that comes clpsest A to your personal opinion. 
1. How useful was the amount of detail in the unit? > 

Not enough' detail to understan4 the unit ' * 



JJnJLt wopld have been clearer with more' detail * < 
^Appropriate amount of detail 

JInit was occasionally too detailed, but this was not distracting 
Too much detail; I was often distracted 



How helpful were the! problem answers ? % 

Sample solutions were too brief; I could not do the intermediate steps 

Sufficient information was given to solve -the problems 

Sample solutions were too detailed; I didn f tfneed them 



3. Except for fulfilling the prerequisites, how much did you use other sources (for 
example, instructor, friends, or other books) in order to uriderstand the unit? 

. A Lot SomewhatVt<* A Little L Not at all 

4. How long was this unit in comparison to the amount of time you generally spend on 
a lesson (lecture and homework Assignment) in a typical math or science course? 

Much Somewhat . ^bout Somewhat " ' Much 
Longer Longer the Same * Shorter ^ Shorter 

5. . Were any of the following parts of the unit crihf using or distracting ? (Check 

as many as apply •) , . , * v 

« P rerequisites - ' 

/ Statement of skills and concepts (objectives) 



^Paragraph headings 
^Examples 

^Special Assistance Supplement (if present) 
•Other, please explain_ 



6. Were any of the following parts of the unit particularly helpful? (Check as many 
as ajfply.) 

Prerequisites 1 4 

Statement of skills and concepts (objectives) 



^Examples 
^Problems 

^Paragraph headings 
JTable of Contents 

^Special"' Assistance Supplement (if present) 
Other \ please explai n , 



Please describe "anything inthe unit that you did not ^particularly like. 



Please describe anything that you found particularly helpful. (Please use the back of 
this sheet if you need more space.) 



